From lab to life: Dr Fu Hong's vision for a world without misaligned eyes

Dr Fu Hong, Associate Professor in the Department of Mathematics and Information Technology (MIT), and her team successfully developed the world’s first prototype machine for an Intelligent Ocular Misalignment Measurement System in 2021.  

Leveraging advanced AI algorithms, Dr Fu’s invention offers a novel solution to the diagnosis of strabismus. By objectively tracking eye movements, the AI-enabled system provides a fast and accurate diagnosis within minutes, reducing costs by 86% compared to manual measurements.

Its ingenious concept and potential contribution to the medical field led to the system winning two international gold awards in 2023. More importantly, this novel technology is highly regarded by ophthalmologist who specialise in strabismus, and the Hong Kong Association of Squint and Double Vision Sufferers.

In the latest edition of FLASS FORWARD, Dr Fu shares what motivated her to delve into her research and how her work can improve the lives of strabismus patients. She also recalls some of the challenges and memorable moments she faced during her research.

A1: Strabismus (斜視) is a condition characterised by the misalignment of the eyes, where the two eyes point in different directions. People have six muscles that work together to control eye movements, ensuring both eyes point in the same direction. For people with strabismus, their eye muscles have problems controlling eye movement and can’t maintain normal ocular alignment. Strabismus can be a congenital or an acquired abnormality.

If one eye is in focus and the other is not, this creates a discrepancy in the brain, resulting in double vision and, consequently, strenuous headaches. To tackle this, the brain might order one of the eyes to shut down and ignore signals sent from it completely. In the long-run, people can develop amblyopia, commonly known as lazy eye. In serious cases, one of the eyes will permanently lose its function.

Without both eyes functioning properly, people cannot develop 3D vision. Without 3D vision, individuals struggle to judge distances accurately. This can lead to difficulties in everyday tasks such as pouring liquids, climbing stairs, catching a ball, crossing streets and driving safely, and so forth. 3D vision develops before the age of eight. People might suffer from weak 3D vision for the rest of their lives if their visual abnormality is not detected early enough.

Strabismus can affect individuals of all ages, but it is most commonly observed in children. As it affects a person’s appearance, strabismus has a significant impact on an individual’s confidence and social life. The eye disease affects approximately 4% of the global population. People with the condition may experience crossed eyes or a misaligned gaze, double vision, difficulty with depth perception, and uncoordinated eye movements. Timely diagnosis is often delayed by limited medical resources and high diagnostic costs. Without proper and timely intervention, it could lead to total blindness in the worst-case scenario.

A2: When my older son was a student at kindergarten, his teacher suspected his eyes had misalignment issues. I took him to an optometry clinic where he was diagnosed with strabismus. My son was fortunate in that his case was not very serious and could be rectified by eye movement training under the instruction of an optometrist. However, the treatment process was costly and laborious.

In each therapeutic session, the optometrist conducted various assessments to diagnose my son’s strabismus in order to decide on the appropriate treatment options. While these tests may differ in details, they all emphasise on how to measure focusing performance and eye movement. Among them, the cover test is the most popular method used to measure ocular misalignment. This test involves covering one eye while the other uncovered eye is observed by the optometrist to identify any misalignment. The assessment is fundamentally a manual one which is time-consuming.

I was thinking at the time, why not use AI to create an intelligent automated system for effective ocular misalignment assessment and diagnosis? I believed such a system could help mitigate the shortage of eye professionals and provide an objective method with highly granular measurements. In the end, together with my team members, I invented a system integrated with Artificial Intelligence (Al) analytics and deep-learning technologies. The invention helps optometrists to conduct clinical assessments of strabismus and enables timely diagnosis to prevent delays in treatment and lasting damage to the patient’s vision.  

The system can be synchroised with data-logging devices to provide immediate feedback.

A3: Initially, my team developed an alpha version of the strabismus evaluation system using digital video to capture a patient’s eye movements during the cover test. This system employed a six-stage algorithm, including eye region extraction, iris boundary detection, keyframe detection, pupil localisation, deviation calculation, and strabismus evaluation. The assessment system demonstrated promising results by achieving an accuracy of over 91% in the horizontal direction and over 86% in the vertical direction.

My research team then developed a comprehensive contour-eye image recognition model in 2021 to further enhance accuracy and efficiency. This model addressed the limitations of previous research by incorporating a set of parametric curves to create a comprehensive eye model. The new model is trained with a deep neural network to enable it to evaluate the fitness of the Contour-Eye image. In addition, evolutionary computation was utilised in the new model to search for the best-fitting curve set. The evaluation of the algorithm using 2,498 eye images from 50 patients confirmed its accuracy, surpassing other relevant studies in contour-eye fitting.

 

The strabismus detection system has developed into a machine prototype which is  equipped with an easy-to-use user interface. Due to the simplicity of the interface, individuals can quickly learn how to operate it. The person being diagnosed only needs to sit in front of the device and look into the digital targets in front of them. In less than two minutes, an accurate diagnostic report can be generated.

The new technology not only offers a solution for measuring ocular misalignment, it addresses the shortage of eye professionals and provides an objective method with highly precise measurements. Given the convenience and ease of operation of the device, I am confident that it will be widely used in the diagnosis of strabismus in future. 

A4: We are fortunate that the innovation has won several international accolades. In 2023, it received two gold medals, including a Gold Award at the 48th International Exhibition of Inventions Geneva and a Gold Award at the 8th International Invention Innovation Competition in Canada, iCAN, where it also earned the Jury's Choice Award. Additionally, at the Hong Kong Digital Asset Society (HKDAS) Innofront 2024, our team garnered the Innovative Award and Excellence Award for related research in computer vision algorithms with applications in healthcare and education.

Due to its significant commercial potential, the strabismus assessment system has received considerable interest from industry stakeholders. Consequently, the knowledge and technology associated with the system were successfully transferred as a non-exclusive patent valued at over HK$1.10 million to a leading high-tech biomedical company based in Beijing, China.

A5: Firstly, we needed to use a structured approach to collect and process high-quality eye movement data. Based on the large number of photographs taken of human ocular movements, we then trained AI to understand how such movements reflect different eye conditions. Based on this data, we developed an AI algorithm to mimic the diagnosis.

Throughout the project, our team needed to learn from optometrists and ophthalmologists about the basic structure of the eyes and the various eye conditions related to strabismus. We needed to acquire new terminology and knowledge in optometry that was outside the AI field. Researchers need to prepare for such challenges if they engage in cross-disciplinary research.

After acquiring the patent for the technology, the mainland biomedical enterprise developed it into a prototype machine for Intelligent Ocular Misalignment Measurement in 2021. After a series of tests and trials, the enterprise has readied the production line for mass production. In the meantime, the enterprise has partnered with a number of hospitals in various cities across mainland China to conduct a series of trial runs using the machine prototype.  

The research team at EdUHK and the Beijing-based enterprise have set our sights on selling the products to hospitals on the mainland. That means we need to obtain approval from the National Medical Products Administration (NMPA,  國家藥品監督管理局), China’s equivalent of the Food and Drug Administration (FDA) in the US. The approval process is very rigorous, requiring preclinical testing, clinical trials, and post-trial monitoring to ensure safety and effectiveness. Each stage requires detailed documentation and compliance with regulatory standards, which can be time-consuming.

It is understandable that the use of medical devices, in hospitals or elsewhere, is subject to stringent regulatory requirements. While we need to be patient and wait for the results, we are confident that our technology is mature and safe to pass through the tests, and will finally get the approval from the concerned authorities.   

I believe that when the new technology is fully commercialised, it could improve the lives of strabismus patients and enable better ocular care, thus creating a valuable impact on our society and community.

A6: The government has been promoting collaboration among industry, academia and research (產學研) sectors so as to transform R&D outcomes from higher education institutions into industrial products. There are hundreds and thousands of steps involved in the whole process from fundamental theoretical research, research and development, technology transfer, and final product launch.

Leading in this project has deepened my understanding of the whole chain of research and development, knowledge transfer and production, which involves the industry, academic and research sectors. On one hand, in order to create the best AI algorithm to understand eyeball movements and related ocular misalignments, my team needs to have a full understanding of the ocular misalignment problem, in addition to their AI expertise. This is an example of vertical integration of knowledge.  

Horizontally, we also need to understand how to transfer academic discoveries into immediate products at the research and development stage before moving on to mass-production by the industry sector. We need to understand the commercial concerns of the industry partner about mass production, and the concerns of the medical profession. This is the biggest takeaway for me.

A7: I thank the Knowledge Transfer Office for assisting the team to develop a prototype and for supporting us to take part in various international competitions. The office also helped the invention with applying for a patent and licence. I was overjoyed when I learnt that the invention was finally patented which reflects its real market value.

I sincerely hope that our invention will pass through the approval procedures by the NMPA and be produced in bulk finally, so that children can have their vision abnormalities detected early. In areas where there are not enough medical professionals, the invention will be particularly useful as it can quickly diagnose strabismus and conduct extensive clinical checks for potential patients. Data collected by the invention will help to devise the best treatment plans for strabismus patients and support further clinical research.

When developing the misalignment measurement system, we needed to collect data at the optometry clinic managed by another university. Students and staff there showed high appreciation of our invention. Some patients taking part in the trials also told me that they were glad to know there is such an invention that helps diagnose eye illnesses. Knowing that delayed intervention to strabismus might result in underdeveloped 3D vision, an unfavourable outlook and, in some cases, total blindness, I really feel joyful that our invention can help improve the situation.                                         

A8: The performance of athletes depends on how well their body coordination is, that is how well their hands, eyes and body can respond and act in synchronisation. A major role of a coach is to train athletes how to improve their coordination. By employing similar technology, we can use AI to analyse eye, hand and body movements, including studying how an athlete’s eye movement is related to their athletic performance, and give feedback to the athletes for continual improvements.

We have been working with related parties to adapt and incorporate the strabismus measurement technology into the Boccia athletes' training system to improve eye-tracking accuracy during visualisation.

We are working together with the Beijing-based company on further cooperation to refine the strabismus misalignment measurement technology for broader use in the sporting field. We have successfully secured funding from the Innovative and Technology Fund by the Hong Kong government for such a project.

Statistics show that 5%-6% of school-aged children suffer from developmental coordination disorder (DCD). It is a motor learning disability that may seriously affect the physical and mental health of affected children. Behaviour analysis of children helps explore the mechanism of DCD and develop more accurate diagnosis and better treatments. With adequate adjustment, the strabismus measurement technology can be used to investigate the behavioural patterns of children with DCD in gross movement using a visual-motor tracking system.

Please click to view a video about the Intelligent Ocular Misalignment Measurement System.